System and method for multimodal, motion-aware radar imaging

ABSTRACT

A radar imaging system to reconstruct a radar reflectivity image of a scene including an object moving with the scene, includes an optical sensor to track the object over a period of time including multiple time steps to produce, for each of the multiple time steps, a deformation of a nominal shape of the object, and an electromagnetic sensor to acquire snapshots of the scene over the multiple time steps to produce a set of radar reflectivity image of the object with deformed shapes defined by the corresponding deformations of the nominal shape of the object. The system also includes a processor configured to determine, for each of the multiple time steps using the deformation determined for the corresponding time step, a transformation between the radar reflectivity image of the object acquired by the electromagnetic sensor at the corresponding time step and a radar reflectivity image of the object in the prototypical pose, and to combine the radar reflectivity images of the object with deformed shapes transformed with the corresponding transformations to produce the radar reflectivity image of the object in the prototypical pose.

TECHNICAL FIELD

The present disclosure relates generally to radar systems, and moreparticularly radar imaging of an object moving in a scene.

BACKGROUND

High-resolution radar imaging is a requirement in a variety of remotesensing applications. For example, radar reflectivity imaging is used invarious security, medical, and through-the-wall imaging (TWI)applications. Whereas the down-range resolution is mostly controlled bythe bandwidth of the transmitted pulse, the cross-range (azimuth)resolution depends on the aperture of the radar sensors. Typically, thelarger the aperture, the higher the image resolution is, regardless ofwhether the aperture is physical (a large antenna) or synthetic (amoving antenna). Currently, the increase of the physical size of antennaleads to a significant increase of the cost of the radar system. To thatend, a number of radar imaging systems use synthetic-aperture methods toreduce the size of the antennas and the cost of radar imaging. Forexample, synthetic-aperture radar (SAR) and inverse SAR (ISAR) use therelative motion of the radar antenna and an object in the scene toprovide finer spatial resolution with comparatively small physicalantennas, i.e., smaller than the antennas of beam-scanning radars.

However, the small size of physical antennas of radar systems makes thetracking of deformable moving objects difficult. Specifically, trackingobjects exhibiting arbitrary motion and deformation requires trackingsensitivity with minimum resolution greater than the resolution of thephysical antennas resulting in an impractical cost of the radar imagingsystem. To that end, conventional radar and/or other electromagnetic oracoustic wave imaging systems require the object to be standing still inthe scene or moving in a very controlled rigid motion. Even for therigid motion, conventional radar imaging systems require a challengingtracking step to estimate the motion parameters of the moving objectusing only the radar data, before a radar image can be formed, see,e.g., Martorella 2014. (Martorella, M. (2014). Introduction to inversesynthetic aperture radar. In Academic Press Library in Signal Processing(Vol. 2, pp. 987-1042). Elsevier.)

Therefore, there is a need for radar imaging systems and methodssuitable for tracking an arbitrarily deformable moving object, usingradar antennas of comparatively small aperture.

SUMMARY

It is an object of some embodiments to provide a radar imaging systemthat allows tracking the motion of an object even if the object isdeformable and the motion is not rigid. It is a further object of someembodiments to provide such a radar imaging system that can reconstructa radar reflectivity image of the object moving in a scene with aresolution greater than a resolution governed by practically sizedphysical antennas of electromagnetic sensors acquiring radarreflectivity images. It is another object of some embodiments to providea radar imaging system suitable for airport security applicationallowing a person to freely move in front of radar imaging system whilereconstructing a radar reflectivity image of the person.

Some embodiments are based on recognition that one of the reasonpreventing electromagnetic sensors of a radar imaging system to track amoving object is a resolution of the electromagnetic sensing governed bya physical size of the antennas of the sensors. Specifically, for thepractical reasons, the size of the antennas of the radar imaging systemcan allow to estimate only coarse image of the object at each time step.Such a coarse image can be suitable to track an object subject to rigidand finite transformation, but can fail to recognize arbitrarilynon-rigid transformation typical for the motion of a human.

Some embodiments are based on recognition that a radar imaging systemcan jointly use measurements of a scene acquired over multiple timesteps. Such a system of measurements can be used to improve theresolution of the radar reflectivity image beyond a resolution governedby the size of the antennas of the radar imaging system. However, whenthe object is moving over time, at different time steps, the object canbe located at different position and can have a different shape causedby the non-rigid motion. Such a dislocation and deformation of theobject make the system of measurements ambiguous, i.e., ill-posed, anddifficult or impractical to solve.

Some embodiments are based on realization that for a number ofapplications, it is sufficient to determine a radar reflectivity imageof an object at some prototypical pose, not necessarily at a currentpose that object has at a current instance of time. For example, forsome security applications, the prototypical pose of a person isstanding, with the hands extended upwards or sideways. The objectarranged in the prototypical pose has a nominal shape that can change,i.e., deform, as the object moves.

Not rigidly moving objects can have different shape at differentinstances of time. To that end, at different time steps there can bedifferent deformations of the shape of the object with respect to itsnominal shape and different transformations of a radar reflectivityimage observed by the radar imaging system with respect to a radarreflectivity image of the object arranged in the prototypical pose. Someembodiments are based on understanding that the transformations of theradar reflectivity images can be inferred from tracking the deformationof the shape of the object. The transformations to the radarreflectivity image of the object arranged in the prototypical poseprovide the commonality to the system of measurements, such that knowingthe transformations can help to resolve the ambiguity of the system ofmeasurements and to solve the system of measurements with methodssimilar to the methods synthetic-aperture image reconstruction with astationary object. However, as discussed above, the resolution of theradar reflectivity images govern by the physical size of the antennas ofthe electromagnetic sensors can be insufficient to track the movingobject.

Some embodiments are based on another realization that a transformationof the radar reflectivity image can be inferred using object tracking ina different modality. For example, some embodiments use a realizationthat the transformation can be seen as a subsampled permutation thatpermutes and/or removes different points of the radar reflectivity imageof the object in the prototypical pose in dependence of the deformationof the nominal shape of the object in the prototypical pose. However,for the purpose of radar image reconstruction, the reflectivity of eachpoint in the radar reflectivity images does not change with thedeformation of the object. Furthermore, the points of in the radarreflectivity image correspond to points in a reflectivity image of adifferent modality, such as an optical reflectivity. Thus, areflectivity of a different modality, such as an optical reflectivity,can capture the same motion and deformation of the object as the radarreflectivity.

Some embodiments are based on recognition that optical sensors, such asmonochrome or color or infrared video cameras, or depth cameras or acombination thereof, are cheaper than electromagnetic sensor withcomparable resolution. Hence, an optical sensor can be used for trackingthe motion of the target, even if the target is deformable and themotion is not rigid. Further, some embodiments are based on anotherrecognition that in a number of applications where radar imaging ofdeformable objects is necessary and useful, the object is movingsufficiently close and visible to the radar imaging system such thatoptical sensors can provide sufficient accuracy for tracking. To thatend, some embodiments are based on realization that by aiding the radarreconstruction using the optical motion tracking, the radar imagingsystem can be able to image very complex target objects that are moving.

An example where the target is clearly visible is security applications,in which people walk in front of a scanning system, e.g., in an airport.In some airport security scanners require subjects to be standing in aspecific pose to be scanned for prohibited items. The scanning systemaccording to one embodiment allows the subjects (which are thedeformable moving objects, such as humans) to simply walk through thescanner while they are scanned, without any need to stop.

Accordingly, one embodiment discloses a radar imaging system configuredto determine a radar reflectivity image of a scene including an objectmoving with the scene. The radar imaging system includes an opticalsensor to track the object over a period of time to produce, for eachtime step, a deformation of a nominal shape of the object. As usedherein, the nominal shape of the object is a shape of the objectarranged in a prototypical pose.

The radar imaging system also includes at least one electromagneticsensor, such as a mmWave sensor, a THz imaging sensor, or a backscatterX-Ray sensor, or combinations thereof, to acquire snapshots of theobject over the multiple time steps. Each snapshot includes measurementsrepresenting a radar reflectivity image of the object with a deformedshape defined by the corresponding deformation. In various embodiments,the object can have a non-rigid motion, such that at least two differentsnapshots include radar reflectivity images of the object with differentdeformed shapes.

The radar imaging system also includes a processor to determine, foreach time step using the deformation determined for the correspondingtime step, a transformation between the radar reflectivity image of theobject in the snapshot and a radar reflectivity image of the object inthe prototypical pose. After determining the transformation, theprocessor determines a prototypical radar reflectivity image of theobject in the prototypical pose such that, in each snapshot, theprototypical radar reflectivity image transformed with the correspondingtransformation matches each of the radar reflectivity images of theobject. In such a manner, the cooperative operation of optical andelectromagnetic sensor allows to track the moving object with theresolution sufficient for reconstructing the radar reflectivity image ofthe object in a prototypical pose.

Accordingly, one embodiment discloses a radar imaging system toreconstruct a radar reflectivity image of a scene including an objectmoving with the scene. The system includes an optical sensor to trackthe object over a period of time including multiple time steps toproduce, for each of the multiple time steps, a deformation of a nominalshape of the object, wherein the nominal shape of the object is a shapeof the object arranged in a prototypical pose; at least oneelectromagnetic sensor to acquire snapshots of the scene over themultiple time steps to produce a set of radar reflectivity image of theobject with deformed shapes defined by the corresponding deformations ofthe nominal shape of the object, wherein at least two different radarreflectivity images include the object with different deformed shapes;and a processor to determine, for each of the multiple time steps usingthe deformation determined for the corresponding time step, atransformation between the radar reflectivity image of the objectacquired by the electromagnetic sensor at the corresponding time stepand a radar reflectivity image of the object in the prototypical pose;and to combine the radar reflectivity images of the object with deformedshapes transformed with the corresponding transformations to produce theradar reflectivity image of the object in the prototypical pose.

Another embodiment discloses a method for reconstructing a radarreflectivity image of a scene including an object moving with the scene,wherein the method uses a processor operatively connected to an opticalsensor and an electromagnetic sensor, wherein the processor is coupledwith stored instructions implementing the method, wherein theinstructions, when executed by the processor carry out steps of themethod. The method includes tracking, using the optical sensor, theobject over a period of time including multiple time steps to produce,for each of the multiple time steps, a deformation of a nominal shape ofthe object, wherein the nominal shape of the object is a shape of theobject arranged in a prototypical pose; acquiring, using theelectromagnetic sensor, snapshots of the scene over the multiple timesteps to produce a set of radar reflectivity images of the object withdeformed shapes defined by the corresponding deformations of the nominalshape of the object, wherein at least two different radar reflectivityimages include the object with different deformed shapes; determining,for each of the multiple time steps using the deformation determined forthe corresponding time step, a transformation between the radarreflectivity image of the object acquired by the electromagnetic sensorat the corresponding time step and a radar reflectivity image of theobject in the prototypical pose; and combining the radar reflectivityimages of the object with deformed shapes transformed with thecorresponding transformations to produce the radar reflectivity image ofthe object in the prototypical pose.

Yet another embodiment discloses a non-transitory computer readablestorage medium embodied thereon a program executable by a processor forperforming a method, wherein the processor is operatively connected toan optical sensor and an electromagnetic sensor, the method includestracking, using the optical sensor, the object over a period of timeincluding multiple time steps to produce, for each of the multiple timesteps, a deformation of a nominal shape of the object, wherein thenominal shape of the object is a shape of the object arranged in aprototypical pose; acquiring, using the electromagnetic sensor,snapshots of the scene over the multiple time steps to produce a set ofradar reflectivity image of the object with deformed shapes defined bythe corresponding deformations of the nominal shape of the object,wherein at least two different radar reflectivity images include theobject with different deformed shapes; determining, for each of themultiple time steps using the deformation determined for thecorresponding time step, a transformation between the radar reflectivityimage of the object acquired by the electromagnetic sensor at thecorresponding time step and a radar reflectivity image of the object inthe prototypical pose; and combining the radar reflectivity images ofthe object with deformed shapes transformed with the correspondingtransformations to produce the radar reflectivity image of the object inthe prototypical pose.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a radar imaging system 100 to determinea radar reflectivity image of a scene 105 including an object movingwith the scene according to some embodiments.

FIG. 2 shows a schematic of a relative tracking used by some embodimentsusing the optical sensor.

FIG. 3 shows a block diagram representing the relationship betweentransformation of a shape of an object in the optical reflectivity imageand a transformation of the object in the radar reflectivity image usedby some embodiment for radar reflectivity image reconstruction.

FIG. 4 shows a schematic of illustrating some principles employed byvarious embodiments for radar reflectivity image reconstruction.

FIG. 5 shows a schematic of dual-grid representation 500 of an objectaccording to some embodiments.

FIG. 6 shows a schematic capturing the motion of the object using thedual-grid representation according to one embodiment.

FIG. 7 shows a schematic capturing the transformation of the objectcaused by its motion using the dual-grid representation according to oneembodiment.

FIG. 8 shows a schematic of an electromagnetic sensor, such as a radar,acquiring the radar reflectivity image according to some embodiments.

FIG. 9 shows a schematic of reconstruction of a radar reflectivity imageaccording to one embodiment.

FIG. 10 shows an example of the motion and deformation of the object infront of the optical and radar sensors at each snapshot used by someembodiments.

FIG. 11 shows a schematic of the tracking performed by the opticalsensor using the example of FIG. 10.

FIG. 12 shows a hardware diagram of different components of the radarimaging system 1200 in accordance with some embodiments.

DETAILED DESCRIPTION

FIG. 1 shows a block diagram of a radar imaging system 100 to determinea radar reflectivity image of a scene 105 including an object movingwith the scene according to some embodiments. The radar imaging system100 includes at least one optical sensor 110 configured to acquire anoptical reflectivity images of the scene and at least oneelectromagnetic sensor 120 configured to acquire a radar reflectivityimages of the scene. Examples of the optical sensor 110 include one orcombination of an optical camera, a depth camera, and an infraredcamera. Examples of the electromagnetic sensor 120 include one orcombination of a millimeter wave (mmWave) radar, a terahertz (Thz)imaging sensor, and a backscatter X-ray sensor.

The optical sensor 110 is configured to track the object in the sceneover a period of time including multiple time steps to produce, for eachof the multiple time steps, a shape of the object at a current timestep. In various embodiments, the optical sensor 110 determines theshape of the object as a deformation 115 of a nominal shape of theobject. For example, the nominal shape of the object is a shape of theobject arranged in a prototypical pose typically known in advance.

The electromagnetic sensor 120 is configured to acquire snapshots of thescene over the multiple time steps to produce a set of radarreflectivity images 125 of the object with deformed shapes defined bythe corresponding deformations of the nominal shape of the objectdetermined by the optical sensor. Notably, due to the movement of theobject in the scene, at least two different radar reflectivity imagesinclude the object with different deformed shapes.

The radar imaging system 100 includes at least one processor 130. Theprocessor 130 is configured to determine 140, for each of the multipletime steps using the deformation 115 determined for the correspondingtime step, a transformation 145 between the radar reflectivity image ofthe object acquired by the electromagnetic sensor at the correspondingtime step and a radar reflectivity image of the object in theprototypical pose. Notably, because the object is moving in the scene,different radar reflectivity images can have different transformations.In one embodiment, the optical and the radar reflectivity images aresynchronized, e.g., taken concurrently at corresponding time stepsand/or with a predetermined time shift, such that the transformation ofa radar reflectivity image is determined using the deformation produceby the corresponding optical reflectivity image synchronized or acquiredat the same time step.

Some embodiments are based on recognition that a radar imaging systemcan jointly use the measurements of a scene acquired over multiple timesteps. Such a system of measurements can be used to improve theresolution of the radar reflectivity image beyond a resolution governedby the size of the antennas of the radar imaging system. When the objectis moving or deforming over time, at different time steps, the objectcan be located at different position and can have a different shapecaused by the non-rigid motion. Such a dislocation and deformation ofthe object make the system of measurements ambiguous, i.e., ill-posed,and difficult or impractical to solve. However, the embodiments can,instead, exploit the diversity introduced by the motion by determiningand using the transformations between each radar reflectivity image andthe common image, i.e., the radar reflectivity image of the object inthe prototypical pose. Thus, the embodiments can to jointly usemeasurements of a scene acquired over multiple time steps to produce theradar reflectivity image of the object in the prototypical pose.

To that end, the processor 130 is configured to combine 150 the radarreflectivity images of the object with deformed shapes transformed withthe corresponding transformation to produce the radar reflectivity imageof the object in the prototypical pose. For example, the combinationdetermines the radar reflectivity image of the object in theprototypical pose such that in each snapshot, the prototypical radarreflectivity image transformed with the corresponding transformationmatches each of the radar reflectivity images of the object.

Some embodiments are based on recognition that one of the reasonpreventing electromagnetic sensors of a radar imaging system to track amoving object is a resolution of the electromagnetic sensing governed bya physical size of the antennas of the sensors. Specifically, for thepractical reasons, the size of the antennas of the radar imaging systemcan allow to estimate only coarse image of the object at each time step.Such a coarse image can be suitable to track an object subject to rigidand finite transformation, but can fail to recognize arbitrarilynon-rigid transformation typical for the motion of a human.

Some embodiments are based on recognition that optical sensors, such asmonochrome or color or infrared video cameras, or depth cameras or acombination thereof, are cheaper than electromagnetic sensor withcomparable resolution. Hence, an optical sensor can be used for trackingthe motion of the target, even if the target is deformable and themotion is not rigid. Further, some embodiments are based on anotherrecognition that in a number of applications where radar imaging ofdeformable objects is necessary and useful, the object is movingsufficiently close and visible to the radar imaging system such thatoptical sensors can provide sufficient accuracy for tracking.

Some embodiments are based on realization that for a number ofapplications, it is sufficient to determine a radar reflectivity imageof an object at some prototypical pose, not necessarily at a currentpose that object has at a current instance of time. For example, forsome security applications, the prototypical pose of a person isstanding, with the hands extended upwards or sideways. The objectarranged in the prototypical pose has a nominal shape that can change,i.e., deform, as the object moves.

Some embodiments are based on another realization that the opticalsensor can track the object in the scene in relative, rather than anabsolute mode. For example, instead of determining a pose and/or anabsolute current shape of the object in the current snapshot of thescene, the optical sensor can determine the relative shape of the objectas a deformation a nominal shape of the object. The embodiments arebased on realization the deformation of the nominal shape of the objectdetermined by the optical sensor can be used to reconstruct radarreflectivity image of the object in the prototypical pose, e.g., havingthe prototypical shape.

FIG. 2 shows a schematic of a relative tracking used by some embodimentsusing the optical sensor. The embodiments track a moving object, e.g., aperson, by determining at each time instant a deformation 220 of anominal shape 210 of the object resulting in the current deformed shape230 of the object. Some embodiments are based on realization that thedeformation 220 indicative of transformation of an object in an opticalreflectivity image is also indicative of the transformation of theobject in the radar reflectivity image. To that end, by aiding the radarreconstruction using the optical motion tracking, the radar imagingsystem according to some embodiments can be able to image very complextarget objects that are moving.

FIG. 3 shows a block diagram representing the relationship betweentransformation of a shape of an object in the optical reflectivity imageand a transformation of the object in the radar reflectivity image usedby some embodiment for radar reflectivity image reconstruction. Forexample, some embodiments use a realization that the deformation 220 canbe seen as permutations of locations and/or intensities of the points ofthe optical reflectivity image of the object having the nominal shape.Specifically, the deformation 220 can be seen as a subsampledpermutation 310 that permutes locations and/or removes different pointsof the optical reflectivity image of the object having the nominalshape. In addition, the deformation 220 can be seen as a permutation 320of intensities of those permuted points.

Some embodiments are based on realization that the transformation 350 inthe radar reflectivity image can be seen as a subsampled permutationthat permutes and/or removes different points of the radar reflectivityimage of the object in the prototypical pose in dependence of thedeformation of the nominal shape of the object in the prototypical pose.To that end, the permutation 310 can be preserved 330 to arrive on thetransformation 350.

On the other hand, for the purpose of radar image reconstruction, thereflectivity of each point in the radar reflectivity image does notchange with the deformation of the object. Furthermore, the points ofthe radar reflectivity image correspond to points in a correspondingoptical reflectivity image. Thus, a reflectivity of a differentmodality, such as an optical reflectivity, can capture the same motionand deformation of the object as the radar reflectivity. On the otherhand, the intensity of the points in the radar reflectivity image arethe same between snapshots and can be estimated from the measurementsduring reconstruction. Beyond its use in estimating the deformation ofthe object, the intensity of the optical reflectivity and itspermutation may not be useful in determining the radar reflectivityimage and may be discarded 340.

FIG. 4 shows a schematic of illustrating some principles employed byvarious embodiments for radar reflectivity image reconstruction.Specifically, some embodiments are based on understanding that a set ofradar reflectivity images 410 of an object in the same pose butdifferent position can be used to reconstruct 420 the radar reflectivityimage. Typically, the resolution of the images 410 is less than theresolution of the image 420. Such an approach allows reducing the costof the electromagnetic sensors used to acquire the radar reflectivityimages. Typically, in this situation, the object only exhibitstranslation and rotation relative to the radar over multiple time stepsover which the radar reflectivity images 410 are acquired.

Some embodiments are based on realization that this approach can bemodified for the situation when the object in the radar reflectivityimages 440 deforms to different shapes, i.e., the object is non-rigidand moving in the scene. Specifically, the embodiments use thetransformation 350 to transform the radar reflectivity images 440 withthe objects having different shapes into the radar reflectivity images450 with the objects having the same shape, i.e., a nominal shape of anobject in a prototypical pose. Those radar reflectivity images 450 canbe reconstructed to produce the reconstructed radar reflectivity image460 of the object in the prototypical pose. As before, the image 460 canhave a higher resolution than the resolution of the images 440.

For example, one embodiment selects the nominal shape of the objectarranged in the prototypical pose as a shape of the object in one of theimages 440, e.g., an image 441. In such a manner, the transformation 350between the radar reflectivity image 441 of the object having theselected shape and corresponding image 451 of the object in theprototypical pose is an identity transformation. Additionally oralternatively, in one embodiment, the information indicative of thenominal shape of the object arranged in the prototypical pose isreceived, e.g., via an input interface.

Some embodiments are based on realization that the representation of thesame object can be defined in two distinct discretization grids, i.e.,frames of reference and coordinate/indexing systems in imaging theobject. The first grid is the discretization on the object itself,typically, but not necessarily, defined on a prototypical pose of theobject. For example, a typical prototypical pose for a human isstanding, with the hands extended sideways. The grid indexes points onthe human and deforms as the human moves. However, the points do notmove relative to the human anatomy. E.g., the point at the tip of thethumb always has the same index, no matter where the thumb might bepositioned relatively to the rest of the body. The second grid is thediscretization of the space in front of the radar sensor. This grid istypically radial or Cartesian (i.e., rectangular) and is fixed relativeto the sensor.

As the target moves in front of the radar, the first grid deforms—sinceit always conforms to the shape of the target—and occupies differentpositions on the second grid. In other words, in some embodiments, themap of the points of the first grid to points on the second grid is asubsampled permutation, corresponding to the first grid deformation,which can be obtained from the tracking algorithm and the opticalsensor(s) used for motion tracking. More generally, the map of therepresentation from the first grid to the second grid is a lineroperator that is obtained by the tracking algorithm and the opticalsensors, in combination with the radar properties. In some embodiments,if not a subsampled permutation, this map may be a blurred or filteredsubsampled permutation, or a blurred or filtered subsampled rearrangingof coordinates in a continuous space, that is subsequently discretized,thus allowing for points moving in-between grid locations.

This dual-grid representation of an object allows to streamline therelationship between the optical and radar reflectivity images to reducethe computational cost of the radar reflectivity image reconstruction.Using knowledge of this deformation, and multiple snapshots it ispossible to reconstruct the target image using techniques similar toISAR, but sufficiently modified to take the deformation into account.This modification is challenging and requires newly developed solver, inorder to account for possible occlusions and/or tracking error.

FIG. 5 shows a schematic of dual-grid representation 500 of an objectaccording to some embodiments. For example, a deformable object 510, ahuman in this example, is in a prototypical pose. To construct a radarreflectivity image in the prototypical form, a grid 520 can be definedon the prototypical pose of the object. In other words, the first grid520 of the dual-grid representation is a prototypical grid thatdiscretizes the object itself. For example, the grid in the image hasgrid positions 530 indexed as 1, 2, . . . , N. The radar reflectivity ofeach point in the grid 540 is denoted as x₁, x₂, . . . , x_(N). Thereare several ways to index the prototypical grid, but in general they canalways be mapped to a sequential grid, as shown in the figure. Any gridindexing approach can be used in embodiments of the invention.

The second grid of the dual-grid representation is a radar grid thatdiscretizes the scene itself. For example, in one embodiments the secondgrid is a rectangular (Cartesian) grid 550. However, other grids, suchas a radial one are also be used by different embodiments. As with theprototypical grid, there are several ways to index the radar grid usedby different embodiments. For example, in the embodiment shown in FIG.5, the radar grid is indexed using Cartesian coordinates 560. Theelectromagnetic sensor, i.e., radar 570, and/or individual radartransceivers 580 have positions inside the radar grid, at knownlocations.

FIG. 6 shows a schematic capturing the motion of the object using thedual-grid representation according to one embodiment. FIG. 6 illustratesthe object in the prototypical pose 600, as well as the object's pose infront of the radar at a single snapshot 650. The object's pose in frontof the radar can be described by a deformation 640 of the first grid torepresent the deformation of the object in the second grid. Thereflectivity of the object in the radar grid is observed by the radar670 and its individual transceivers 680 according to a radar forwardoperator related to the hardware of the radar.

As described above, the deformation of the radar grid can be inferred bythe optical sensor, since the transformation of the object is the sameboth in the radar and the optical modalities. Thus, the optical sensorcan observe the optical image deformation, and infer the deformation ofthe prototypical grid to the radar grid.

FIG. 7 shows a schematic capturing the transformation of the objectcaused by its motion using the dual-grid representation according to oneembodiment. This embodiment determines each transformation as asubsampled permutation that permutes locations of some points of theradar reflectivity image of the object in the prototypical pose andremoves different points of the radar reflectivity image of the objectin the prototypical pose in dependence of the deformation of the nominalshape of the object in the prototypical pose.

Specifically, the deformation is a subsampled coordinate permutation745, i.e., a transformation that maps the indices in the coordinatesystem of the prototypical grid to the indices of the radar grid. Thus,the reflectivity of the object observed by the radar is a simplepermutation with erasures that maps 760 the reflectivity of the objectin the prototypical grid 610 to the reflectivity of the object in theradar grid, consistent with the object's pose.

More generally, in some embodiments, the reflectivity image of thedeformed object in the radar grid is a linear transformation of thereflectivity image of the object in the prototypical pose, that can bedescribed asz=Fx,  (1)where x is the radar reflectivity image of the object in theprototypical pose and z is the radar reflectivity image of the deformedobject in the radar grid.

FIG. 8 shows a schematic of an electromagnetic sensor, such as a radar,acquiring the radar reflectivity image according to some embodiments.The radar 870 includes transceivers 880 (or separate transmitters andreceivers) which are used to sense the scene. In particular, one or moreof the transceivers transmit a pulse 820 to the scene. The pulses areabsorbed by or reflect of the object 850, according to the reflectivityof the object. The reflected pulses 830 are acquired by one or moretransceivers 880. The pulse transmission and acquisition, is controlledby a radar control, pulsing and acquisition system 890, which maycontrol the pulse shape and timing, as well as which transceiverstransmit and which receive.

The system is also configured for acquiring the signals that thetransceivers produce in response to the received pulses. The systemoutputs data y 895 which represent recordings of the pulse reflections.These recordings are samples of the reflections or a function of them,such as demodulation, filtering, de-chirping, or other pre-processingfunctions known in the art.

The acquired data y are linear measurements of z, the radar reflectivityimage of the deformed object 850 in the radar scene, through the radaracquisition function, also known in the art as the forward operatordenoted here using A. Thus, the acquired data for a single snapshot areequal toy=Az=AFx.  (2)

If the radar system has a sufficient number of sensors and a bigaperture, then the data y may be sufficient to recover z, the radarreflectivity image of the object in the deformed pose. However,recovering the image in high resolution would require a large andexpensive radar array. Furthermore, in particular deformations, parts ofthe object might not be visible to the array, which can make their radarreflectivity not recoverable, irrespective of the radar array size.

For that reason, a radar imaging system of some embodiments acquireseveral snapshots of the radar image, under different deformationsy _(i) =Az _(i) =AF _(i) x,  (3)where i=1, . . . , T is the index of the snapshot, and T is the totalnumber of snapshots. In various embodiments, the only change betweensnapshots is the deformation of the object, and, therefore, thedeformation F _(i) of the radar reflectivity image. In some embodiments,the radar forward operator can be different when, for example, differenttransducers are used to acquire each snapshot. This is accommodated inthe above model by modifying (3) to include a snapshot-dependent forwardoperator:y _(i) =A _(i) z _(i) =A _(i) F _(i) x.  (4)

It is evident that (4) describes (3) when A=A_(i) for all i.

Some embodiments reconstruct the radar reflectivity image of the objectin the prototypical pose by combining the radar reflectivity images ofthe object with deformed shapes transformed with the correspondingtransformations. For example, using multiple snapshots, thereconstruction problem becomes one of recovering x from

$\begin{matrix}{{\begin{bmatrix}y_{1} \\\vdots \\y_{T}\end{bmatrix} = {\begin{bmatrix}{A_{1}{\overset{\_}{F}}_{1}} \\\vdots \\{A_{T}{\overset{\_}{F}}_{T}}\end{bmatrix}x}},} & (5)\end{matrix}$which, assuming the F _(i) are known, can be performed using, e.g.,least squares inversion. Some embodiments also impose additionalregularization constraints on reconstructing image x, such as sparsityor smoothness, by expressing x in some lower-dimensional basis of asubspace or a large dictionary, i.e, x=Bh, where B is a low dimensionalbasis or a dictionary, and h is a set of coefficients eitherlower-dimensional than x or sparse. Alternatively, or additionally, someembodiments impose low total variation structure on x, i.e., sparsity inits gradient. All these regularization constraints can be imposed byregularizing using different techniques.

Some embodiments determine the transformation F _(i) of the radarreflectivity image from the deformation of an image acquired in adifferent modality, such as an optical image. In other words, by using ahigher resolution modality it is possible to infer the physicaldeformation of the object. Since the reflectivity of each point in theobject's radar reflectivity image does not change with the position ofthe point, i.e., with the deformation of the object, the deformationinferred by the sensor of the different modality can be used to inferthe deformation of the object's radar reflectivity.

Optical sensors, such as monochrome, color, or infrared cameras recordsnapshots of the reflectivity of objects as they move through a scene.Using two or more of these cameras, placed at some distance apart, it ispossible to determine the distance of each point of the object from eachcamera, known in the art as depth of the point. Similarly, depth camerasuse the time-of-flight of optical pulses or structured light patterns todetermine depth. By acquiring the optical reflectivity and/or the depthof the object as it moves, there are techniques in the art to track thepoints of the object, i.e., to determine, in each snapshot, thedeformation of the objects from the deformation of the optical or thedepth image. Determining this deformation is possible in the art, eventhough the optical reflection of the object changes with deformation dueto lighting, occlusion, shadowing and other effects.

Thus, an optical sensor, such as a camera or a depth sensor, can be usedto infer the deformation F _(i) in each snapshot. The optical sensoracquires a snapshot of the object at the same time instance as the radarsensor acquires a snapshot of the radar reflectivity image, as describedin (4). This radar reflectivity image can then be used to track thedeformation of the object in order to reconstruct its radar reflectivityimage. In some embodiments, the optical sensor might acquire snapshotsat different time instances than the radar sensor. The deformation ofthe object at the time instance that the radar acquires the snapshot iscan then be inferred using multiple optical snapshots, using techniquesknown in the art, such as interpolation or motion modeling.

FIG. 9 shows a schematic of reconstruction of a radar reflectivity imageaccording to one embodiment. In this embodiment, the radar imagingsystem includes one or more electromagnetic sensors, such as radararrays 910, and one or more optical sensors 920. The object 930, forexample a human, moves and deforms in front of the radar and the opticalsensors, while the sensors acquire snapshots. The data acquired by theoptical sensor are processed by an optical tracking system 940, whichproduces a tracking of the object and its deformation 950 from snapshotto snapshot. The optical tracking system 940 also maps the opticaldeformation to the object's prototypical pose, i.e., determines themapping F _(i) for each snapshot. This mapping is used together with thedata acquired in each radar snapshot to reconstruct 970 the radarreflectivity image of the object 980. The reconstructed radarreflectivity image may be represented in the prototypical pose by thesystem, by may be converted and represented in any pose and with anymodifications suitable to the system or its user, for example tohighlight parts of the image for further examination 990.

In such a manner, the radar imaging system includes an optical trackingsystem including the optical sensor to produce each deformation toinclude an optical transformation between points of an opticalreflectivity image including the object in the deformed shape and pointsof a prototypical optical reflectivity image including the object in thenominal shape. The processor of the radar imaging system determines thetransformation as a function of the optical transformation.

FIG. 10 shows an example of the motion and deformation of the object infront of the optical and radar sensors at each snapshot used by someembodiments. In this example, a human 1090 walks in front of thesensors. The sensors obtain snapshots at different time instances, withthe object in different pose. For example, in FIG. 10, at each snapshot,the human is at a different position in front of the sensors, walkingfrom left to right, and at a different pose, depending on the timing ofeach snapshot relative to the stride of the human.

FIG. 11 shows a schematic of the tracking performed by the opticalsensor using the example of FIG. 10. Notably, there is a snapshot basedone-to-one correspondence between the deformation of the shape of theobject in the optical reflectivity image and correspondingtransformation of the radar reflectivity image.

Each point on the human 1100 is tracked by the camera at each timeinstant, and then mapped to the corresponding point in the prototypicalpose 1190. Each point might or might not be visible in some snapshots.For example, points on the right shoulder 1110, right knee 1120, orright ankle 1130 might always be visible, while points on the left hand1150 might be occluded when the hand in behind the body and not visibleto the sensors 1160. The tracking creates correspondences 1180 betweenpoints in different snapshots and the corresponding point in theprototypical image. The correspondences are used to generate F _(i). Ifa point is not visible to the sensor at some particular snapshot, e.g.,1160, the F _(i) does not map this point to the radar grid, i.e., thecorresponding column of the operator contains all zeros. In that sense,F _(i) subsamples the prototypical radar image to only map the pointsthat are visible in the particular snapshot, as determined by theoptical sensor.

FIG. 12 shows a hardware diagram of different components of the radarimaging system 1200 in accordance with some embodiments. The radarimaging system 1200 includes a processor 1220 configured to executestored instructions, as well as a memory 1240 that stores instructionsthat are executable by the processor. The processor 1220 can be a singlecore processor, a multi-core processor, a computing cluster, or anynumber of other configurations. The memory 1240 can include randomaccess memory (RAM), read only memory (ROM), flash memory, or any othersuitable memory systems. The processor 1220 is connected through a bus1206 to one or more input and output devices.

These instructions implement a method for reconstructing radarreflectivity image of the object in the prototypical pose. To that end,the radar imaging system 1200 can also include a storage device 1230adapted to store different modules storing executable instructions forthe processor 1220. The storage device stores a deformation module 1231configured to estimate the deformation of the object in each snapshotusing measurements 1234 of the optical sensor data, a transformationmodule 1232 configured to obtain the transformations of the radarreflectivity images which is an estimate of F _(i) the opticaldeformation; and reconstruction module 1233 configured to solve for x inEquation (4) above using the estimate F_(i), in place of the true F_(i), and optionally applying regularization, as described above. Thestorage device 1230 can be implemented using a hard drive, an opticaldrive, a thumb drive, an array of drives, or any combinations thereof.

The radar imaging system 1200 includes an input interface to receivemeasurements 1295 of the optical and electromagnetic sensors. Forexample, in some implementations, the input interface includes a humanmachine interface 1210 within the radar imaging system 1200 thatconnects the processor 1220 to a keyboard 1211 and pointing device 1212,wherein the pointing device 1212 can include a mouse, trackball,touchpad, joy stick, pointing stick, stylus, or touchscreen, amongothers.

Additionally or alternatively, the input interface can include a networkinterface controller 1250 adapted to connect the radar imaging system1200 through the bus 1206 to a network 1290. Through the network 1290,the measurements 1295 can be downloaded and stored within the storagesystem 1230 as training and/or operating data 1234 for storage and/orfurther processing.

The radar imaging system 1200 includes an output interface to render theprototypical radar reflectivity image of the object in the prototypicalpose. For example, the radar imaging system 1200 can be linked throughthe bus 1206 to a display interface 1260 adapted to connect the radarimaging system 1200 to a display device 1265, wherein the display device1265 can include a computer monitor, camera, television, projector, ormobile device, among others.

For example, the radar imaging system 1200 can be connected to a systeminterface 1270 adapted to connect the radar imaging system to adifferent system 1275 controlled based on the reconstructed radarreflectivity image. Additionally or alternatively, the radar imagingsystem 1200 can be connected to an application interface 1280 throughthe bus 1206 adapted to connect the radar imaging system 1200 to anapplication device 1285 that can operate based on results of imagereconstruction.

Some embodiments are based on recognition that in some cases, theestimate F_(i) of F _(i) contains errors. The errors can be modeled as F_(i)=E_(i)F_(i), where E_(i) has similar structure as F_(i), but is moreconstrained. In other words, E_(i) is also a subsampled permutation or amore general operator which allows, e.g., blurring. However, becauseE_(i) models the errors in the motion tracking, and the motion trackingis in general correct, the mapping that E_(i) performs is only allowedto displace points a little bit away from the position the estimateF_(i) placed them in. In summary, F_(i), which is computed from themotion tracking part of the system, places the target grid inapproximately the right position, and E_(i) makes small corrections tothis placement.

To that end, in some embodiments, the processor adjusts eachtransformation with a local error correction and determines concurrentlythe radar image of the object in the prototypical pose and each localerror correction. For example, the processor determines concurrently theradar image of the object in the prototypical pose and each local errorcorrection using one or combination of alternating minimization,projections, and constrained regularization.

Those embodiments are based on recognition that the motion trackingerror E_(i) is generally unknown. Otherwise, if the error is known, itwould have been trivial to correct the motion error. Thus, the radarimaging system also estimates E_(i) from the snapshots, in order tocorrect the error. In other words, in some embodiments, the processor ofthe radar imaging system is configured to solve

$\begin{matrix}{{\begin{bmatrix}y_{1} \\\vdots \\y_{T}\end{bmatrix} = {\begin{bmatrix}{A_{1}E_{1}F_{1}} \\\vdots \\{A_{T}E_{T}F_{T}}\end{bmatrix}x}},} & (6)\end{matrix}$where all the E_(i) are unknown, in addition to x.

Various embodiments use different methods to recover the error, such asalternating minimization, or lifting to a higher space, by estimatingthe outer product of the E_(i) and x, and imposing low-rank structure onthe resulting object. Note that the restricted structure of E_(i) can beimposed using constraints and/or regularization.

As above, assuming the radar has obtained sufficient snapshots, theestimate can be computed, and x be recovered. To that end, the processorsolves for x and the E_(i) in (6) above using the estimate F_(i) inplace of the true F _(i), and applying constraints and regularization asnecessary, and as described above. Whether the tracking system isassumed to perform errors, or not, i.e., whether we correct for theerror or not, the overall system recovers x, which is the image ofobject of interest, indexed by the grid in the prototypical pose.

The above-described embodiments of the present invention can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. Such processorsmay be implemented as integrated circuits, with one or more processorsin an integrated circuit component. A processor may be implemented usingcircuitry in any suitable format.

Also, the embodiments of the invention may be embodied as a method, ofwhich an example has been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments.

Use of ordinal terms such as “first,” “second,” in the claims to modifya claim element does not by itself connote any priority, precedence, ororder of one claim element over another or the temporal order in whichacts of a method are performed, but are used merely as labels todistinguish one claim element having a certain name from another elementhaving a same name (but for use of the ordinal term) to distinguish theclaim elements.

Although the invention has been described by way of examples ofpreferred embodiments, it is to be understood that various otheradaptations and modifications can be made within the spirit and scope ofthe invention.

Therefore, it is the object of the appended claims to cover all suchvariations and modifications as come within the true spirit and scope ofthe invention.

What is claimed:
 1. A radar imaging system to reconstruct a radarreflectivity image of a scene including an object moving with the scene,comprising: an optical sensor to track the object over a period of timeincluding multiple time steps to produce, for each of the multiple timesteps, a deformation of a nominal shape of the object, wherein thenominal shape of the object is a shape of the object arranged in aprototypical pose; at least one electromagnetic sensor to acquiresnapshots of the scene over the multiple time steps to produce a set ofradar reflectivity image of the object with deformed shapes defined bythe corresponding deformations of the nominal shape of the object,wherein at least two different radar reflectivity images include theobject with different deformed shapes; and a processor to determine, foreach of the multiple time steps using the deformation determined for thecorresponding time step, a transformation between the radar reflectivityimage of the object acquired by the electromagnetic sensor at thecorresponding time step and a radar reflectivity image of the object inthe prototypical pose; and to combine the radar reflectivity images ofthe object with deformed shapes transformed with the correspondingtransformations to produce the radar reflectivity image of the object inthe prototypical pose.
 2. The radar imaging system of claim 1, whereinthe processor determines each transformation as a subsampled permutationthat permutes locations of some points of the radar reflectivity imageof the object in the prototypical pose and removes different points ofthe radar reflectivity image of the object in the prototypical pose independence of the deformation of the nominal shape of the object in theprototypical pose.
 3. The radar imaging system of claim 2, wherein thetransformation is the discretization of a blurred or filtered subsampledrearranging of coordinates in a continuous space.
 4. The radar imagingsystem of claim 2, the subsampled permutation preserves intensities ofthe points in permuted locations.
 5. The radar imaging system of claim1, further comprising: a tracking system including the optical sensor toproduce each deformation to include an optical transformation betweenpoints of an optical reflectivity image including the object in thedeformed shape and points of a prototypical optical reflectivity imageincluding the object in the nominal shape, wherein the processordetermines the transformation as a function of the opticaltransformation.
 6. The radar imaging system of claim 1, wherein thenominal shape of the object arranged in the prototypical pose isselected as a shape of the object in one of an optical or radarreflectivity image, such that the transformation between the radar imageof the object having the selected shape and the radar reflectivity imageof the object in the prototypical pose is an identity transformation. 7.The radar imaging system of claim 1, further comprising: an inputinterface to accept information indicative of selection of the nominalshape of the object arranged in the prototypical pose; and an outputinterface to render the radar reflectivity image of the object in aselected pose, including the prototypical pose.
 8. The radar imagingsystem of claim 1, wherein the processor determines the radarreflectivity image of the object in the prototypical pose that, whentransformed by the transformation determined for each time step andacquired by the radar acquisition function, is consistent with themeasurements at all of the multiple time steps.
 9. The radar system ofclaim 8, wherein, for the determining the radar reflectivity image ofthe object in the prototypical pose, the processor regularizes using oneor combination of sparsity in basis or dictionary, smoothness under abasis or dictionary transformation, and a total variation of thereflectivity image.
 10. The radar imaging system of claim 1, wherein theprocessor adjusts each transformation with a local error correction anddetermines concurrently the radar image of the object in theprototypical pose and each local error correction.
 11. The radar imagingsystem of claim 10, wherein the processor determines concurrently theradar image of the object in the prototypical pose and each local errorcorrection using one or combination of alternating minimization,projections, and constrained regularization.
 12. The radar imagingsystem of claim 1, wherein the object moving in the scene is a person.13. The radar imaging system of claim 1, further comprising: a pluralityof electromagnetic sensors with a fixed aperture size, wherein theprocessor determines the radar reflectivity image of the object in theprototypical pose by combining measurements of each electromagneticsensor.
 14. The radar imaging system of claim 13, wherein theelectromagnetic sensors are moving according to known motions, andwherein the processor adjusts the transformation of the radarreflectivity image of the object acquired by the electromagnetic sensorat the corresponding time step based on the know motion of theelectromagnetic sensors for the corresponding time step.
 15. The radarimaging system of claim 1, wherein the electromagnetic sensor includesone or combination of a mmWave radar, a Thz imaging sensor, and abackscatter X-ray sensor, wherein the optical sensor includes one orcombination of an optical camera, a depth camera, and an infraredcamera.
 16. The radar imaging system of claim 1, wherein a resolution ofthe radar reflectivity image of the object in the prototypical pose isgreater than resolutions of the radar reflectivity images of the objectwith deformed shapes.
 17. A method for reconstructing a radarreflectivity image of a scene including an object moving with the scene,wherein the method uses a processor operatively connected to an opticalsensor and an electromagnetic sensor, wherein the processor is coupledwith stored instructions implementing the method, wherein theinstructions, when executed by the processor carry out steps of themethod, comprising: tracking, using the optical sensor, the object overa period of time including multiple time steps to produce, for each ofthe multiple time steps, a deformation of a nominal shape of the object,wherein the nominal shape of the object is a shape of the objectarranged in a prototypical pose; acquiring, using the electromagneticsensor, snapshots of the scene over the multiple time steps to produce aset of radar reflectivity images of the object with deformed shapesdefined by the corresponding deformations of the nominal shape of theobject, wherein at least two different radar reflectivity images includethe object with different deformed shapes; determining, for each of themultiple time steps using the deformation determined for thecorresponding time step, a transformation between the radar reflectivityimage of the object acquired by the electromagnetic sensor at thecorresponding time step and a radar reflectivity image of the object inthe prototypical pose; and combining the radar reflectivity images ofthe object with deformed shapes transformed with the correspondingtransformations to produce the radar reflectivity image of the object inthe prototypical pose.
 18. The method of claim 17, wherein the processordetermines each transformation as a subsampled permutation that permuteslocations of some points of the radar reflectivity image of the objectin the prototypical pose and removes different points of the radarreflectivity image of the object in the prototypical pose in dependenceof the deformation of the nominal shape of the object in theprototypical pose.
 19. The method of claim 17, wherein the object movingin the scene is a person.
 20. A non-transitory computer readable storagemedium embodied thereon a program executable by a processor forperforming a method, wherein the processor is operatively connected toan optical sensor and an electromagnetic sensor, the method comprising:tracking, using the optical sensor, the object over a period of timeincluding multiple time steps to produce, for each of the multiple timesteps, a deformation of a nominal shape of the object, wherein thenominal shape of the object is a shape of the object arranged in aprototypical pose; acquiring, using the electromagnetic sensor,snapshots of the scene over the multiple time steps to produce a set ofradar reflectivity image of the object with deformed shapes defined bythe corresponding deformations of the nominal shape of the object,wherein at least two different radar reflectivity images include theobject with different deformed shapes; determining, for each of themultiple time steps using the deformation determined for thecorresponding time step, a transformation between the radar reflectivityimage of the object acquired by the electromagnetic sensor at thecorresponding time step and a radar reflectivity image of the object inthe prototypical pose; and combining the radar reflectivity images ofthe object with deformed shapes transformed with the correspondingtransformations to produce the radar reflectivity image of the object inthe prototypical pose.